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Self-Evaluations and Visiting Committees 

Effects of Quality Assessment in Dutch Higher Education * 



1 Introduction 

/. 7 Quality Assessment in Dutch Higher Education 

Quality management** is an issue in Dutch higher education since the 1980s. Quality assessment 
(or 'qa' for short) of education was introduced on the political agenda as part of the new policy 
of the government, with the policy paper Higher Education: Autonomy and Quality (1985). In 
exchange for a larger measure of administrative autonomy the universities collectively promised 
to retain and enhance their levels of quality in education. Quality assessment then appeared on a 
systematic and nation-wide scale in 1988, when the Association of Universities in the Netherlands 
(vsNU) effected its new responsibility. The vsNU qa system consists of an external Visiting 
Committee Cvc') for each discipline or cluster of study programmes,*** operating nation-wide. 

To prepare for the vc, each study programme is required to write a self-evaluation (we will 
conceive 'self-evaluation' and *self-study' as synonyms). When the vc has visited all study 
programmes in its *area', it writes a report, based on the self-evaluations that were its input and 
on the experience of the visits to the locations. These visits usually last for two intensive days, 
during which the vc talks to representatives of all actors involved in the study programme 
(including students) and at the end of which, based on the self-evaluation and the impressions of 
the visit, a preliminary comment and judgement about the study programme is given by the chair. 
This text, after being commented by the study programme, is included in the report of the visiting 
committee. Tlie comments, recommendations and judgements about the individual study 
programmer are preceded in the report by a chapter on the general state of the art and shared 
problems, etc., in the discipline covered. 

The self-evaluations and the report by the vc are the key documents in this quality spiral, of 
which the second cycle will start in 1994. In 1990 a parallel system of QA was introduced by the 
VSNU counterpart, the HBO Council, for the non-university sector. The vc's judgements and self- 
evaluations are intended, in the first place, to enhance the quality of the study programme; 
accountability to the government and society in general is o ily the second goal (Vroeijenstijn & 
Acherman, 1990). This raises the question to what extent the results of the evaluations are really 
utilised, in other words what are the effects of the existing qa system? 



* A prcvioua vcreion of this paper waa prcscnlcd at the 15* EAIR Fonim in Turku, 15-18 AugUBt,1993. 
** Quality management is defined aa: Hhat aspect of the overall management function that dctcrminei and implements the 
quality policy (intentions and direction of the organization]'. Quality assurance is: *all those planned and systematic actions 
nccessajy to provide adequate confidence . . This usage is in accordance with ISO 8402 (emphasis added). Quality 
assessment is not defined there and will be taken to mean: a systematic examination to detemune whether quality activities 
comply with planned arrangeoKnts and whether the ^product' (the educational process) is implemented effectively and is 
suitable to achieve objectives. 

*** *Study programme' will be used in this paper to designate a course and its organizational setting ('studierichting'). 
The term Misciplinc* will be used here in a broad sense, meaning each area or cluster distinguished in the QA procedures. 
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question whether or not a visitation had taken place. As a result of this selection 35 qa experts on 
the study programme, faculty and institutional level were interviewed. 



2 Theoretical Framework and Hypotheses 



2. / A Conceptual Model of Quality Assessment 

The Dutch quality assessment system may suggest that a linear relatiorship exists between being 
informed about evaluation results and the utilization of these results by individuals or groups 
belonging to higher education organizations. However, empirical research often shows that no use 
is made of evaluation results (among many others: Cooley & Bickel, 1986). The explanations given 
for this non-utilization can be divided into two groups of arguments. One group emphasizes the 
shortcomings of the evaluation or professional failings of the evaluators (Patton, 1978; Creemers, 
1979; Van den Berg, 1987). The second group contends that evaluation often takes place in an 
ambiguous decision-making context: Who makes decisions? For which decisions are evaluation 
results intended? (Ross, 1980; Weiss, 1982; Scheerens, 1987) Whatever the reason, serious doubts 
are raised about the implied linear relationship between knowing evaluation results and utilizing 
them. Therefore, it seems important and useful to determine the effects of evaluations (internal and 
external) on higher education organizations and to look for possible explanations of whether or not 
foUow-up activities are undertaken. 

A framework for this is given by Patton et al. (1977), who distinguish three categories of 
characteristics that could be related to utilization of evaluation results, namely: 
o characteristics of the organization where the evaluation takes place; 
o characteristics of actors: evaluators and decision-makers of the organization; 
o characteristics of the evaluation itself. 

Figure 1 contains a conceptual model in which Patton' s categories can be recognized and that is 
based on a systems theoretical approach, visualizing the relationships among evaluations of the 
quality of teaching and the effects of these evaluations, as well as a cluster of explanatory 
variables. In this figure we see that information about educational characteristics is the input for 
evaluation and that the output consists of evaluation results (analyses and judgements). The extent 



CiuracUriitki 




Evaloation 




Evahiatioa 
Retnlti 


enrolmeot 

students 

teaching staff 

cusiculum 

fitcilities 

graduatca 

other characteristics 




aelf-cvaluatioii 

visiting committee 

other types of 
evaiuatioo 




analyses 
judgements 



1 



fttdkMcki Effects Extent of utilization 



coMiext: Charadecistica of 
organizatioti and acton 



Figure 1 Conceptual model of the quality management process 

G 



4 



Frederiks, Westerheijden & Weusthof 



to which results are being used to improve the quality of education is, for our purpose, the effect 
of the quality management system. The characteristics of actors, together with characteristics of 
the organization, are 'contextual* factors that influence the degree and way of utilization. 

We define the key dependent variable, effect, as any way of utilization of results of the qa 
system. ^Utilization' then is defined as the extent to which the results of an internal or external 
evaluation are handled by a higher education organization. We distinguish between no utilization, 
passive utilization and active utilization. *No utilization' means that evaluation results are neglected 
by a higher education organization. 'Passive utilization' refers to use of results without immediately 
making decisions to change (parts of) the curriculum or the organization, but, for instance, to 
disseminate evaluation reports within the organization, to discuss the results of the process in 
formal settings, or to make recommendations for future changes. 'Active utilization' is defined as 
taking measures on the basis of available evaluation results. One can distinguish a ranking of 
utilization from no utilization to passive and active utilization, but that does not imply a normative 
ranking: active utilization is not necessarily 'better' than passive utilization. It may well be that no 
active utilization is called for, e.g., when the evaluation results are completely positive. Also, 
passive utilization may lead to future changes that are not directly linked (in the consciousness of 
the organization members) to the evaluation, but that would not have occurred without it. Another 
point is that opportunities for change may differ. Not only active utilization is not expected when 
all evaluation results are positive, but also some vcs succeed better in fonnulating 
recommendations that can be used for decision-making than other vcs (Van Lieshout, 1992). 

In connection with the conceptual framework a number of hypotheses are formulated. The 
argumentation for these hypotheses is based on the contingency approach (Child, 1984) and the 
political economics approach (Lieshout, 1989). These two approaches can provide us with explana- 
tions for the relationship between on the one hand characteristics of higher education organizations 
and the actors that operate inside these organizations and on the other hand the extent of utilization 
of the external (vc) and internal evaluation (self-study) results. 

2.2 The Contingency Aoproach 

The basic assumption of the contingency approach is that an effective organization is optimally 
adjusted to (contingent with) specific environmental circumstances. The central question of this 
approach is: under which conditions can an organization function most effectively? Child (1984) 
distinguishes a number of factors which can be conceived as explanations for the sometimes large 
differences between organizations. The most important contingency factors in the context of our 
research are: the environment, the technology, the size and the centralization of an organization. 
We assume that environment, technology, size and centralization are related to the extent of active 
utilization of the internal and external evaluation results. The hypotheses based on this approach 
will now be presented and substantiated. It should be noted that all hypotheses represent partial 
relationships, with the other factors being held constant for the time being (the ceteris paribus 
clause). 

The environment of higher education organizations consists of, among others, potential students 
and employers. Also the financing bodies of higher education institutions, of which the national 
government is the most important, are part of the environment. We conceive the visiting committee 
as another primary environmental factor for a higher education organization. Although it goes too 
far to say that higher education is ftilly determined by its environment, it can be maintained that 
the external influence (labour market and visiting committee) still grows and that the goals of 
higher education institutions are more and more externally conditioned. 
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Because of the serious consequences that a negative judgement of a visiting committee can have 
for a study programme (student enrolments may decrease or finance may be at risk), we expect 
more active utUization of evaluation results on the basis of the vc report than on the basis of the 
self-evaluation report. 

Hypothesis 1 : External evaluations by visiting committees have a higher probability, compared 
to the internal self-evaluations, to result in active utilization of the evaluation report. 

We distinguish two characteristics that refer to the structure of a higher education organization, 
namely the size of the study programume and the extent of centralization. The larger the size of an 
organization becomes, the more the need evolves to write down all kinds of previously unwritten 
rules and informal procedures: empirical research has confirmed that a positive relationship exists 
between the size of an organization and the extent of formalization (Mintzberg, 1979). The 
consequence of larger size may be that decision-making on the basis of evaluation results (active 
utilization) takes longer or even never takes place at all. It can be assumed that larger size of study 
programmes will lead to more formal decision-making processes, with long advice and discussion 
tracks and more possibilities for delaying decisions. 

Hypothesis 2: The larger a study programme, the less likely it is that active utilization of 
evaluation results occurs. 

A negative judgement of the vc might have severe consequences for tiie study programme. In 
anticipation of this potential threat the management may seek to enlarge its control over the study 
programme. In this way optimal conditions can be created for the external peer review, which 
enhances die chances for a positive judgement. Empirical research (Baldridge et al., 1977) shows 
that a positive relationship exists between the presence of an external threat (in this case the vc) 
and centralization. In case of a threat the formal decision-makers will try to get more grip on the 
decision-making process (centralization) and use their increased power to take measures (active 
utilization of the evaluation results) that they conceive as necessary to cope with the external threat. 

Hypothesis 3: The higher the centralization of a study programme, the more likely it is that 
active utilization of evaluation results occurs. 

We can define technology informally as the knowledge about ways of producing (Groot, 1988). 
Scheerens (1987) describes the technology of higher education as 'unclear': it is uncertain how the 
input of the educational process (teachers, students, teaching materials) is exactiy transferred into 
output (graduates). This uncertainty will have a negative impact on the extent of aaive utilization, 
since it is not known what difference it makes to change any single aspect. We have taken the 
availability of data required by the vsNU and hbo Council for the self-evaluations as indicator for 
the uncertainty about technology. 

Hypothesis 4: The higher the availability of data required by vsNU and HBO-Council for the self- 
evaluations, the more likely it is that active utilization of self-evaluation results occurs. 

2.3 The Political Economics Approach 

Apart fi-om the characteristics of organizations we can also identify characteristics of actors withm 
the organization. Especially important in this context are the reputations for power of these actors. 
The political economics approach is based on the principle of maximization of utility. Lieshout 
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(1989) equalizes utility to the concept of power. This results in the proposition of an actor 
maximizing power, whose perceptions of the power (hence reputations for power) of relevant other 
actors, combined with their strivings for maximization of their own power, forms the input for its 
weighing of alternative courses of action. For our purpose, two types of actors can be distinguished 
in higher education institutions, namely, evaluators (the actors that carry out the internal 
evaluations, such as the self-study) and the decision-makers, who have to decide on how to handle 
the evaluation results. 

The reputation for power of the evaluators is decisive for the question whether decision-makers 
take the self-evaluation seriously. If the recommendations are threatening for the status quo and the 
reputation for power of the evaluators is relatively low, the decision-makers have not much to fear 
from not following the advice. 

Hypothesis 5: The lower the reputation for power of evaluators, the less likely it is that the 
results of the self -study are actively used. 

Not only the reputations for power of the evaluators play an important role in determining the 
reaction to evaluations, but also those of the decision-makers. Powerful actors are more likely not 
to use recommendations that might harm the status quo and therefore their interests. 

Hypothesis 6: The higher the reputation for power of decision-makers, the less likely it is that 
active utilization of evaluation results occurs. 

A final important point is related to the overlap between evaluators and decision-makers. If the 
actors who perform the evaluation and the ones that have to decide on the utilization are (partly) 
the same, then the consequence might be a more active utilization, because this is in the interest 
of these evaluators-and-decision-makers. 

Hypothesis 7: The more decision-makers are involved in performing the self-evaluation, the more 
likely it is that results of the self-evaluation are actively used. 



3 Test of the Hypotheses 

In this section we present the findings of the empirical tests of the seven hypotheses. To test these 
hypotheses we have calculated correlations between on the one hand the independent variables 
concerning contingency and power factors and on the other hand the dependent variables ^active 
utilization of the self-evaluation (SE) report' as well as 'active utilization of the vc report*. This 
has been done both for the university and for the non-university sector in Dutch higher education. 
We have used the number of measures taken in response to evaluations as an operationalization of 
the extent of active utilization. 

As we can see in table 1 hypothesis 1 is partly corroborated, namely the proposition about the 
positive influence of environmental pressure (vc) on active utilization; that indeed appears to be 
the case for the universities. To make a comparison between the university and the non-university 
sector possible, active utilization is calculated as the average number of measures following the 
report divided by the average number of recommendations. The contingency hypotheses 2 and 3, 
that relate active utilization to respectively size of the study programme and centralization, are 
faJsified. Interesting in this respect is the significant correlation we have found, concerning the 
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table 1 Correlations between active utilization and characteristics of the organization and actors 



active 



active 



utilization utilization 



HyfjOthesis 



SB report 
universities 



vc report 
universities 



active 
utilization 
S€ report 

HBO 



active 
utilization 
vc report 

HBO 



1 active utilization vc report > 
active utilization SE report 

2 size of study programme 
active utilization (-) 

3 centralization active utilization 




r = .0930 
N = 82 
p<.406 

r = .146 
N = 153 
p<.070 

4 availability of data for SE report r = . 1 509 
active utilization SE report N = 105 

p<.124 

5 reputation for power of r = ,0524 
evaluators active utilization SE N = 1 32 
report p<.551 

6 reputation for power of r = .0963 
decision-makers -» active utilization N = 1 45 
(-) p<.249 

7 involvement of decision-makers r=-.0732 
in SE active utilization SE report N = 1 45 

p<.382 



r=.0683 
N = 106 
p<.487 



5.2/8.6 = 
.60 
N = 31 

r= -.2698 
N = 21 
p<.237 

r = .1490 
N = 30 
p<.432 



r = .0890 
N=40 
p<.585 

r = .2427 
N=51 
p<.086 

r= - .2376 

N=44 

p<.120 

r = .0315 ^r-^SlSO .r=-.1425 
N = 100 H-'l^/r'r N = 29 
p< .756 ^ P<-461 



Shaded background indicates significant correlation (p < .05) 

relationship between size of dve study programme (indicated by the number of first year students) 
and active utilization of the vc report in universities. This correlation turned out to be positive, 
while a negative relation was expected. The most plausible explanation for this unexpected 
phenomenon seems to be that in the response to our questionnaire the sector languages and cultural 
studies was over-represented in the university sample. This sector contains many small study 
programmes and their active utilization was shown to be, on average, relativdy low. This may 
have been caused by the low level of satisfaction with the vcs for language programmes.* So for 
this sector there is a positive rdation between active utilization and size, which because of the 
over-representation of this sector may result in such a positive correlation for the entire university 
population. The remaining hypothesis based on the contingency jqpproach (nuirber 4) is partly 
corroborated. Only for the non-university sector a positive and significant corrdation between the 
data available for the se report and the extent of active utilization of the se report could be 
identified. Further analysis shows that in the university sector the variation in answers tends to be 
less, which may explain why no significant correlation is established. 

The same argument may also apply to the partial corroboration of hypotheses 6 and 7, based 
on the political economics approach. Empirical support f')r the expected negative relation between 



* In the second round, this area of knowledge will be covered by many specialist VCs. instead of the two broad VCs thai 
operated in the first ttxind. A sense of dissatisfaction apparently has been noted in the VSNU too. 
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reputation for power of decision-makers and active utilization (hypothesis 6) has only been found 
for HBOs and then solely for the se report. Possibly the reputation for power of decision-makers 
in the non-university sector Is more important than in the university sector, where regulation of 
decision-making about educational matters is more extensive and formalized and hence the variation 
in reputation for power tends to be less. Furthermore, it seems plausible that reputations for power 
of d^lsion-makers are more important In response to se reports and less so In response to vc 
reports, due to the relatively higher reputation for power of the vc. Further analysis shows that 
decision-makers on the central level of the study programme are (by law) more involved in the self- 
evaluation in universities than in the hbo institutions. We can argue that the variation between 
universities and hbos in degree of formalization of the decision-making processes can also 
constitute an explanation for the falsification of hypothesis 7 for the universities (concerning the 
expected positive relation between involvement of decision-makers in the self-study and the active 
utilization of the se report). Also, we have found no support for the expectation that a higher 
reputation for power of evaluators has a positive impact on the active utilization of the se report: 
hypothesis 5 has been falsified. 

In sum, we found some support for both the contingency as well as the political economics 
hypotheses. However, this support consists only of four partially corroborated hypotheses, while 
the other three hypotheses have been falsified. 

In the next three sections, we shall take a broader look at the research data, which may lead to a 
better insight in the way qa is embedded, and to suggestions for alternative explanations, a subject 
to which we shall return in the final section. But first we shall give an overview of the existing 
quality management activities, followed by a cection about the utilization of the SE and vc reports 
and a section In which we will concentrate on the satisfaction with assessment procedures. The 
universities and the hbo institutions will be presented separately, primarily because the qa system 
in the hbo sector started later than in the university sector, so that the experience with It and the 
time to Implement changes were significantly less than in the university sector. 



4 Quality Management Activities 

4. 1 Quality Management in the Universities 
Internal Evaluations and the Self-Evaluation 

Out of the valid responses* to the survey, 95% of the university study programmes Indicate that 
some kind of explicit quality activity Is conducted within the organization. In most cases (64%), 
tliis includes the VSNU procalure (the self-evaluation and/or the visitation). However, 69% indicate 
that previous to the vsNU procedure some form of evaluation existed already, usually because it 
was the local tradition; special occasions, like governmental budget cuts or externally visible failure 
(unemployment), were not often named as reasons. These earlier evaluations were mostly 
concerned with the study programme as a whole (67%) or large parts (several years) of the 
progranmie (19%), so In extent they can be compared to the self-evaluations. The Information 
resulting ft-om previous internal evaluations could be used (parti^ly or in full) for the vsNU self- 
evaluation in more than 80% of the study programmes. 

Internal quality evaluations (see table 2) mostly focus upon the curriculum, or on the topics 



* Valid UMponscg. i.e., all response* minus 'not applicable' or blank, but including 'do not know'. 
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fashionable at the time, either because of public discussions or because of the financial incentives 
built into the government's funding system, i.e., on efficiency of teaching and on retention rates, 
drop out rates and completion time. Slightly less often the aims and goals of the study programme 
are part of the internal quality activities. But characteristics of the students entering the programme 
are only mentioned in 43% of the cases. This probably reflects the freedom of choice, and the 
almost unlimited right to enrol that students have in the Dutch university system: study programmes 
or institutions have hardly any instruments to influence enrolment. It was not possible to find more 
attention for enrolment matters for study programmes that experience more problems in this field 
than others (low numbers of enrolments, deficient knowledge among entering students, especially 
of mathematics and sciences, etc.). 

Internal quality evaluations are indeed to a high extent internal to the university study 
programmes in the points of view taken into account (see table 3). In (practically) all cases, student 
opinions and staff opinions are taken into account. Views of other stakeholders, like faculty 
managers, alumni, or employers all score less than 50%. 



table 3 Points of view used in quality activities 



opinions of 


universities % 


HBO % 


students 


100 


97 


academic/teaching staff 


99 


95 


study programme or facultv/sector managers 


37 


42 


alumni 


48 


75 


employers/professionals 


25 


73 



How was the self-evaluation conducted: was it something relegated to a single staff member? An 
individual staff member was indeed strongly or very strongly involved in 83% of the cases. 
However, that does not imply it was an individual's study: in practically every study programme 
the study programme conunittee (*studierichtingscommissie'),* the dean responsible for education, 
an ad hoc conunittee, or other actors, were strongly or very strongly involved as well. In 80% of 
the valid responses m ad hoc committee was installed: such 'an intensive evaluation effort 
apparently cannot be performed by the normal organization. Indeed, the amount of time and other 
resources needed to prepare the self-evaluation were often mentioned as urgent problems of the 
present qa system, both in the universities and in the HBO institutions. Ad hoc committees usually 
consist of members of the sttidy programme committee and of academic staff, either as 



» In the »tudy programme committee the academic staff and atudcnts arc rcprcjjcntcd equally. It is the most tmpoititnt 
advisory body for the faculty council and boaixJ regarding educational matters. 
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representatives of the departments (Vakgroepen*), or as individuals. If an ad hoc committee exists, 
it is (very) strongly involved in 91% of the cases. 

As a rule, then, the self-evaluation can be considered to be a collective effort, so that the official 
endorsement by the faculty council and board —as required by the vsnu— is backed by a 
reasonable level of involvement of the study programme as a whole. Moreover, the self-evaluation 
is usually indeed done by 'self: in approximately 17% of the cases some actor from outside the 
study programme was involved. And those outside persons or agencies are, almost without 
exception, part of the larger faculty or of the university —in other words: part of the 'inner circle* 
of ihe environment. 

Institutional Quality Management of Teaching 

Quality management of teaching used to be a responsibility primarily of the faculty or study 
programme level within Dutch universities. And at that level, research was considered to be more 
important both for the individual's career development and for the faculty's prestige. With the 
introduction of the national QA procedure, the role of the institution as a whole in educational 
matters has obtained more emphasis; the vsnu procedure plays an important role in the central 
level quality activities. In most of the institutions involved in the case studies of our project, central 
quality management is 'reactive' rather than 'active' —which is a characterization, but not a 
normative judgement. A 'reactive' policy means that initiatives for quality management and 
enhancement are taken by the faculty or study programme, while the central aciors are mostly in 
a supportive or stimulating role, without a central policy being formulated about the quality of 
teaching itself ('meta-level' policy only). This may have to do with the Continental tradition (Clark, 
1983) of Dutch higher education: institutional management is not strongly developed, most 
competencies regarding teaching reside in the faculty or study programme and in this respect 
certainly the university is a 'loosely coupled' system (Birnbaum, 1988) with at least as many 
(management) cultures as there are faculties (Becher, 1989). 

Still 'reactive', but more 'active' than most, is one university where a central committee 
critically reads all self-evaluations for the vsnu qa procedure and comments on their content, 
before ihey are sent to the vsnu visiting committee. Two of our seven case studies (out of thirteen 
universities in the Netherlands) show an 'active' central quality management of teaching, in that 
institution-wide goals are set. In one of the two, the completion rate is the prime indicator, 
supporting sUidy programmes to take measures to raise these if necessary. In the other, a policy 
of raising completion rates has evolved into a more encompassing quality policy where 
'studiability'* of the curricula has become the central issue. 

4.2 Quality Management of Teaching in Hso Institutions 

Internal Evaluations and the Self-Evaluation 

Quality activities are as frequent in the hbo institutions as in the universities: 95% state to have 
some form of quality management activity, even though in only 21% of the cases this includes the 
self-evaluation for the national qa procedure coordinated by the hbo Council (the qa procedure 
started two years later than for the universities). 

The subjects covered by the internal quality activities (see table 2) closely resemble those in the 
universities (curricula, efficiency, retention rates, aims are all mentioned in more than 80% of the 



* 'Sludiability' ii n neologism intixxiuccd in a report on common problems and solutions to lower the drop 
Dutch higher education (Wijncn et ai, 1992). 
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cases). But the attention given to characteristics of the student intake (69%) by far exceeds the 
percentage for the universities (43%). This may indicate more problems with the students entering 
the programmes, possibly as a consequence of their more heterogerieous backgrounds, or the larger 
possibilities hbo programmes have for selection of their students. 

Also like in the universities, the points of view represented in internal quality evaluations are 
predominantly internal to the study programme (see table 3): students and staff in more than 95% 
of the cases, but institutional managers' opinions in less than half of the cases. An interesting 
difference, however, exists as regards the involvement of the environment of the higher education 
institutions: alumni and employers play a role in approximately three quarters of the cases. 
Appai'ently, the hbo study programmes are more open to their ^clients' than university study 
programmes. The quality definitions used in practice in the hbo sector seem to be more consumer 
oriented. This difference mirrors the vocational or professional character of the hbo programmes 
as opposed to the more academically minded universities. As does the fact that internal quality 
evaluations in hbo are more often a response to changes in the external environment, especially, 
on the input side deficient knowledge of entering students and, on the output side changes in labour 
market demands. 

Due to the later start of the national qa system, in the hbo sector only 21% of the study 
programmes had completed their self-evaluation. This means the *empirical base* for statements 
related to the se is rather small (N = 51). In general the impression emerging about the way the 
self-evaluations are written equals that from the university sector: the bulk of the work is done by 
one or a few individuals, but draft versions are fairly widely discussed, thus involving many more 
actors and ensuring sufficient support for the self-evaluation by the study programme as a whole. 
Here too the more direct links between study programmes and clients become apparent: in some 
cases committees representing employers or professionals are involved (albeit weakly) in the 
making of the self-evaluation, which does not happen in university study programmes. 

Institutional Quality Management of Teaching 

Most of the HBO institutions have come into existence in their present form as a result of a large- 
scale merger operation in the 1980s (see Goedegebuure, 1992). It is, therefore, not surprising to 
find that institution-wide quality management activities originated in recent years too. From the 
four institutions included in the case studies in this project, though, we tentatively conclude that 
the HBO institutions are more ^actively* involved in quality management of teaching than the central 
level actors of the universities. Yet also in the hbo institutions, most of the central level quality 
management activities could be called meta-level policy, because most of the instruments developed 
are to stimulate study programme level actors to give more priority to their quality management. 
In one of the cases included in our project, this is done through management contracts. Another 
interesting initiative in several hbo institutions is the organization of *trial visits': the self- 
evaluation is read by a conmiittee appointed by the institution itself (consisting of institutional staff 
members from other study programmes and central level quality managers, and of employers or 
professional representatives) prior to the official hbo Council visiting committee comes for its visit. 
If this practice spreads further, it might be viewed as the nucleus of an even more self-regulatory 
form of quality management in Dutch higher education. 



5 Utilization of Self-Evaluation and vc Report 

The vc report, together with the self-evaluation with its a'^alyses, self-judgements and 
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recommendations, is the input for the next phase of the process at the study programme level, the 
utilization of the evaluation results (the feedback-part of the loop in Figure 1). In table 4 we present 
a first impression of the effects of the internal and external evaluations at the universities and hbos. 
In the next sections we go into active, passive and no utilization of the reports. 

5. / No Utilization 

One way to conclude there was no utilization has been when neither the self-evaluation report, nor 
the vc report, were discussed officially in any kind of meeting (passive utilization), and no 
measures were taken in connection to the5^^. reports (active utilization). These conditions, in 
conjunction, apply to four cases (two in universities and two in hbos). This means, as far as we 
can observe it, that participation in the national QA system for not more than four study 
programmes (out of a sample of 137) is nothing but a 'ritual dance'. However, a different picture 
emerges if we look at no utilization of the se and the vc report separately. No utilization of the 
SE report occurs in 9 out of 154 cases in universities, and in 6 out of 51 hbo study programmes. 
Passive as well as active utilization of the vc report have been absent in 14 out of 106 university 
study programmes, and in 6 out of 31 hbo cases. Although in this broader definition no utilization 
is more wide-spread than our first measurement indicated, there is no reason for exaggeration. 
Still, the vast majority of study programmes in both sectors utilize at least one of the reports either 
actively or passively in its organizational processes: 'ritual dancing' remains a rarity. 

5.2 Passive Utilization 

As we can see in table 4 the passive utilization of both reports (indicated by the average number 
of actors discussing the reports: the maximum number of actors is seven) is higher for the 
universities than for the hbo institutions. lu both sectors the se rapport leads to more formal 
discussion than the vc report. However, ftirther analysis of our survey data shows that the vc 
report is more discussed by the faculty board (and its equivalent in the non-university sector) than 
the SB report, which is consistent with formal decision-making rules. From the fact that sE reports 
tend to be disseminated rather widely throughout the faculty we can deduce the existence of a 
collective interest in the self-evaluation. This does not imply that usually all academic staff 
members and students receive copies of the report, but they all can get access to the information 
if they want. 

In universities the faculty boards, faculty councils, study programme committees and 
departments discussed both reports in more than 50% of the cases. The reports were also discussed 
in 50% or more of the cases by student representatives, but in only 15% (for the vc report) and 
36% (for the SE report) by individual staff members. In contrast with these findings the passive 



table 4 The extent of utilization of internal and external evaluation results 
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utilization by the equivalent of the faculty council in the hbo sector tends to be lower, but the 
discussion of the reports by teachers higher (50% or more). 

From our interviews we formed the impression that on the institutional level the evaluation 
results are hardly utilized. This is not surprising, since the responsibility for the content of the SE 
report and the follow up of both the internal and external evaluation rests exclusively with the 
faculties. However, in a few cases the faculties were assisted by the institutional level in preparing 
the self-evaluation, but only at their request. Furthermore, although passive utilization (formal 
discussion) of the SE report on the institutional level does in general not occur, the vc reports were 
in most cases discussed on the central level (by the Committee of Deans). This passive utilization 
may also include discussion with the faculty and/or the study programme about the follow up, 
especially when the judgements of the vc were rather negative. 

5,3 Active Utilization 

Ultimately, one of the aims of evaluation is improvement of the quality of the study programme. 
As a rule, the evaluation reports show that something could be improved: out of the 3 1 categories 
presented, the average university study programme that has written a SE report indicates that on 
9.6 points (HBOs: 13.5) some recommendation(s) was (were) made in the self-evaluation report, 
and on 7.8 (hbos: 8.6) in the vc report (see table 4). In table 5 the percentage of study 
programmes that have taken measures on a certain subject following the SE and vc 
recommendations are confronted with the percentage of study programmes that received 
recommendations on that subject. From table 5 it can be noticed that in both universities and hbos 
the most ^popular' subjects in SE and vc reports to make recommendations about relate to the 
content and aims of the curriculum (including study progress). In universities considerable less 
advices are being given regarding the qualifications of teachers and graduates. In the hbo sector, 
on the contrary, qualifications of teachers and graduates get much more attention, especially in the 
self-evaluation. Here facilities (computers, libraries, etc.) are the less 'popular' subject for 
recommendation. 

It is clear from table 4 that the average number of recommendations (both for the SE and the 
vc report) is lower for the universities than for the hbo sector. The figures concerning the average 
number of measures following the reports are more diversified. Following the SE report more 
measures are taken in hbo institutions (7.5 vs. 6.3), while in response to the vc report the average 
number of measures is higher in universities (6.7 vs. 5.2). However, this comparison between the 
university and non-university sector is not quite fair: for comparing the active utilization within the 

table 5 Active utilization per subject in QA reports {%) 
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two sectors it seems appropriate to take the average number of recommendations into account. The 
active utilization, corrected for the average number of recommendations, can be seen in table 1 
(hypothesis 1). Then a different picture emerges: the active utilization in both sectors is higher for 
the vc than for the SE report (though the difference is non-signitlcant for the abo sector). 

But the important question, determining the actual effects, must be: are measures taken as a 
consequence of the evaluation process? There certainly is not a one-tOH)ne (Minear*) relationship 
between recommendations and measures. Indeed, the number of times that recommendations on 
a certain subject in the reports correspond directly with measures on the same subject are very low, 
as we can a^so see in table 5. 

Furthermore, it can be noticed that in universities, following both the SE as the vc report, the 
numbers of measures taken regarding teachers are relatively low. This may imply that the 
professional autonomy of university teachers is too large to impose changes in their way of 
teaching, to stimulate staff development programmes, etc. 

Especially interesting are, of course, explanations of the observed non-linear relationship 
between recommendations and following measures, or in other words, the low degree of active 
utilization. Apart from the explanations given by the hypotheses, of which the pressure of the vc 
and partially (only for hbos) the availability of data for self-evaluation, the power reputation of 
decision-makers and their involvement in the self-evaluation have proven to be significant, we were 
also confronted with possible explanations during our case studies. From analysis of se and vc 
recommendations it became clear that not often an actor responsible for the follow up is mentioned. 
This can lead to a situation in which no one feels responsible for follow up actions and 
consequently no active utilization occurs. A period of time in which measures should be taken is 
only mentioned in a small number of recommendations. The omission of a time frame can cause 
endless postponement of active utilization. 

In our interviews with QA experts the following explanations for not transforming the 
recommendations into actions emerged: 

o the decision-makers do not agree with the content of the recommendations; 
o the advice is still being discussed; 

o the recommendations are inconsistent with the policy of the Ministry of Education and Science; 

o the financial means necessary for realizing the recommendations are insufficient; 

o the subjects of recommendation already had the attention of the study programme and additional 

actions are perceived as unnecessary. 
Another possible explanation for not or hardly utilizing evaluation results can be derived by taking 
differences in valuevS, norms and traditions between disciplines into account. These differences in 
disciplinary cultures can be quite substantial (see, e.g., Becher, 1989) and may influence the extent 
of utilization. In some disciplines there may exist a more positive attitude towards evaluation or 
the utilization of evaluation results than in other disciplines. Analysis of variance of our survey data 
indicate empirical support for this assumed relation between disciplinary cultures and the extent of 
utilization. Further research is necessary if we want to go more into detail about the exact nature 
and direction of this relation. 

The measures taken are, as a rule, not very drastic (see table 4). On the contrary, in both 
sectors and both following the self-evaluation and the vc report, the median class of answers is 'not 
far-reaching' (the lowest of the five possibilities for answering) for all subjects except the master*s 
(HBO: bachelor's) and propaedeutic programme. The subjects ^student enrolment*, 'graduates' and 
^teachers' have the lowest averages for drastic measures. The percentages of drastic measures 
(average for all categories in a study programme four or more out of five) are higher, both in 
universities and in HBo institutions, for measures taken following the SE report. This seems logical 
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if we assume that as a rule drastic measures are already taken in anticipation of the visitation. 

A last point we want to make in this section relates to active utilization on the institutional level . 
As has been said before, the utilization of the evaluation reports is the responsibility of the faculties 
and study programmes. Therefore, it may not seem surprising that on the instftutional level hardly 
any actions are being taken as a result of the evaluations. Notwithstanding this, we noted in our 
interviews that some universities and hbos have introduced or are developing a sort of control- 
system. The aim of such a control-system is that the institutional level gets more insight into the 
handling of the vc report by the faculties. This may imply that the faculty informs the Board of 
the iastitution (e.g., in the annual report on education) about the measures that are being taken in 
response to the vc reports and also discusses the follow up in regular meetings with the Board. 



6 Satisfaction with the QA System 

6. 1 Overall Satisfaction 

Included in our questionnaire was also a question about the appreciation by the study programme 
about the different types of evaluations. On a scale from 1 tot 10 (the usual rating scale in Dutch 
education: I being the minimum and 10 the maximum) the respondents could give their judgement 
about the internal quality management, the visitation and the self-evaluation. Respondents were also 
asked to motivate their judgement. The marks that the respondents gave, were taken as an indicator 
for the degree of satisfaction with the elements of the qa system. This can be considered both as 
a side effect of the present round of the vsnu and hbo Council systems and as a first indication 
for the willingness of the study programmes to co-operate in the second round of evaluations. 
Apparently, tlie median class of answers being seven, the study programmes are fairly satisfied 
with the existing qa system(see table 6). Of course, this does not mean that no changes are wished 
for or that there are no variations within the populations (between sectors and disciplines) or 
concerning the different elements of qa. At this place we want to point at the differences between 
the total response and the so called Visited' response (the study programmes that were visited at 
the time of our survey or that were in the process of visitation, viz., that had finished their self- 
evaluation). It can be noted from table 6 that, while in universities the scores for the total and the 
*visited' response hardly differ, in the hbo sector the Visited* study programmes are more satisfied 
about QA than the total response. However, in comparing these averages we must be careftil, 
because the non-university response is considerably lower than the response of universities. In the 
next sections we will go more into detail about the satisfaction with the internal quality 

tabic 6 Satisfaction of respondents with different types of evaluation 
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management, the self-evaluation and the visitation. 
6.2 Satisfaction with Internal Quality Management 

There is little difference in .«b;tisfaction about the internal quality management between the two 
sectors: hbos are slightly more satisfied than universities ('visited response'). Further analysis 
(correlations) shows that study programmes are more satisfied if they use judgements of the faculty 
board and of students in their internal evaluations (both sectors) and if they take characteristics of 
student enrolment into account (only universities). For the hbo sector we found a positive and 
significam (p< .05) correlation between satisfaction and the utilization of judgements of alumni. 

In their motivations respondents in both sectors indicated that regular evaluations and feedback 
to teachers, regular meetings between teachers and departments, participation of students, a small 
size of the study programme, utilization of information systems, modularization of courses, a 
critical self-evaluation and the external pressure of the visitation, contributed to a positive attitude 
towards the internal quality managemem. Obstacles for internal evaluations are also mentioned: 
financial problems, shortages of teachers and the fact that evaluations take a lot of time. A few 
respondents from hbo institutions point at the risk that evaluations are perceived as a charge on 
the autonomy of teachers and have doubts about the appropriation of quantitative evaluation 
methods for small study programmes. From the comments of respondents we got the impression 
that many study programmes in the non-university sector are still in the starting phase of quality 
management. 

6 3 Satisfaction with the Self-Evaluation 

The satisfaction with self-evaluation, as for internal quality management, hardly differs between 
universities and hbo institutions. Analysis of causal relations between satisfaction with SE and other 
variables, derived from the survey, show different types of significant correlations between the two 
sectors University study programmes tend to be more positive towards sE if they already have a 
tradition of evaluation and if the se report has been formally discussed by the faculty board and 
council Study programmes in the non-university sector are more satisfied if in the se report results 
of previous evaluations have been used, the report has been discussed by the teachers and the 
students committee, the number of recommendations in the report is higher and if more measures 
have been taken following recommendations on student enrolment and facilities. 

Positive Judgements are motivated by pointing out that the self-evaluation increases the 
awareness of the importance of quality management, enables the study programme to get a 
thorough overview of 'the state of the art' concerning educational matters and enhances the chance 
on a critical reflection of the educational process. Negative conclusions in the report seem to be 
accepted better if teachers are more involved and have more influence on the follow up. Many 
respondents who are negative about the self-evaluation motivate this by pointing at the heavy 
burden that the making of the se report imposes on the study programme. Also the fact that some 
reports are made by one or just a few people and consequenUy lack the necessary support of larger 
groups within the faculty is a reason for dissatisfaction. Some respondents admit that the external 
pressure of the vc may stimulate the completion of the self-evaluation, but at the same time entails 
the risk that the se relates too much to the framework and references of the vc, thereby loosing 
the importance of the self-evaluation as an instrument for the internal quality management out of 
sight. Furthermore, the utilization of data of institutional information systems is sometimes 
problematic as well as support from the central level in constructing the self-evaluation. 

Not surprisingly then, the QA experts that were questioned in the interviews propose changes 
for the second round that are aimed at a larger involvement of teachers in writing the report, the 
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explication of the preferences of the vc and more attention for the internal function of the SE as 
an instrument for internal quality management. 

6.4 Satisfaction with the Visitation 

The most significant deviation in table 6 that draws our attention is the relatively low appreciation 
of the visitation in universities, both compared to the satisfaction with the other elements of qa. as 
compared to the higher scores for the visitation in the non-university sector. An analysis of the 
scores per discipline shows that this is partially caused by the negative judgement (5.6) of the vc 
in the university discipline languages and cultural studies. In this discipline many study 
programmes were dissatisfied about the broad range of .study programmes clustered under a few 
visiting committees. From further analysis several statistically significant correlations between 
satisfaction and other variables emerged. Study programmes of both sectors have a more positive 
attitude towards the vc if a tradition of evaluation exists. In universities a smaller size of the study 
programme, discussion of the vc report by the faculty council and the study programme 
committee, more recommendations about graduates and following drastic measures on this subject, 
contributed to higher satisfaction. Surprisingly, the only correlations that could be found in the non- 
university sector relate to a lower number ot recommendations concerning facilities and a lower 
number of total measures taken following the vc report. It seems that hbos react more negatively 
to measures taken following the vc report. However, because of the low response in the non- 
university sector we have to be cautious widi this conclusion. 

As motivations for a positive attitude towards the visitation the importance of assessments by 
external experts, the stimulation of a 'quality culture' and the fact that recommendations of the vc 
often support desirable changes, are mentioned. Factors that contribute to a negative attitude are 
that recommendations are often not very precise, reports are sometimes inconsistent, comparisons 
of not comparable study programmes are being made, visitations sometimes coincide with 
reorganizations or holidays, the negative influence that educational political views of the vc can 
have an independent assessment and the belief that if the self-evaluation is carried out properly, 
the vc has little more to add. 

All experts, with one exception, that were questioned in the interviews held the opinion that the 
present QA system should be maintained. This does not mean that changes are not desirable: most 
pleas were made for a less intensive procedure; the procedures take too much time now. 
Furthermore, the preferences of the vc should be more explicated, recommendations must be 
formulated more precisely and in the second round more attention should be given to the follow 
up. University QA experts like to see a less prominent role for retired professors as members of 
the vc and a more important role for foreign experts. Several respondents from the HBO sector 
point at the importance of having more educational scientists in the committee. 



7 Conclusions 

Our first, and main, conclusion is that quality management of teaching is an issue that has obtained 
much more attention than before in Dutch higher education. In a previous research pro=?ct of the 
university sector of Dutch higher education, before the policy changes took effect, one of the 
present authors found that internal quality management was a rare thing (Weusthof, 1989). The 
outcomes of that project suggest that self-evaluations mostly took place at the lower levels of 
universities (teacher evaluation, course unit evaluation) and that the results of these evaluations 
were hardly used in a structured way in faculty decision-making and planning processes. Now, to 
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the contrary, many university and non-university study programmes or faculties have special 
committees or specially appointed staff members for quality management of education, and many 
more have written a self-evaluation: quality of education certainly has gained an important place 
on the agenda of decision-makers. Another remarkable difference between this study and the earlier 
research project concerns the content of the self-evaluation. Seemingly, as a consequence of the 
introduction of the vsNU and hbo Council qa systems not only characteristics of the educational 
process (e.g., didactic qualities of teachers) have become an object of evaluation, as proved to be 
the case in the previous research project, but also input and output characteristics of education (for 
example, giving information to potential students, acquiring data about labour market perspectives 
of graduates). In general we can say that a 'quality culture' may be putting out its roots in the 
universities and hbo institutions. Involvement of the central level actors of higher education 
institutions is less marked, which is in accordance with the procedures set up by the vsnu and the 
HBO Council, for these are focused at the study programme level. But other factors, often stemming 
from the Continental mode of governance, can be mentioned here as well; they could explain the 
predominant *meta-lever policy approach at the institutional level (stimulating study programmes 
to take their quality management seriously, rather than formulating an institution-wide quality of 
teaching policy). 

Second, quality is not only on the agenda, but also something is done about it: measures are 
taken in connection with the self-evaluations and the vc reports. However, we cannot say that the 
large amount of resources invested immediately leads to an equally large iknpiovement of the 
quality of education: measures are not taken in response to every recommendation, nor are the 
measures taken drastic. This is not meant as a criticism of the present system of quality assessment: 
the effects of a 'quality culture' cannot and should not be expected to be immediate and large. 
Quality of education, however important, is only one of a multitude of issues in the institutions for 
higher education; it will take time to grow above the level of the roots it is putting out now; less 
far-reaching measures may have large effects in fiiture years. Furthermore, taking measures does 
not necessarily lead to improvement of education. The relation between taking measures and 
observing improvement is obscure. This originates in the complexity of the education process. 
There are many factors in this process that are very difficult to control. More important still is the 
lack of knowledge about cause and effect relationships in education; taking measures seems to be 
a matter of trial and error and of imitating behaviour. 

Third, the level of satisfaction about the implementation of the Dutch quality management 
system within the institutions is fairly high. This is quite remarkable taking into account that higher 
education organizations are often described as relatively autonomous organizations with little 
inclination to appreciate outside scrutiny. From this point of view the implementation of the present 
quality management system can be seen as a break-in into the ivory tower. Considering the 
relatively high level of satisfaction, the higher education institutions certainly do not shut the doors 
in the face of this attempt to break in. The details of the process may be important in this smooth 
introduction, e.g., ownership and coordination by the institutions themselves, and a rather indirect 
link with (financial) sanctions (see on this matter also: Van Vught & Westerheijden, 1993). The 
level of satisfaction is also remarkable in view of the frequent remarks about how heavy a burden 
it is to write a self-evaluation. From our interviews it seems doubtftil whether the same enthusiasm 
can be mobilized again for the second round of visits. 

Finally, we have to conclude that our effort to explain differences in utilization from a 
contingency theory perspective and from a political economics perspective have largely failed. Even 
though no 'linear' processes of self-evaluation, visitation, recommendations and measures were 
expected, it proved to be more difficult than envisioned to capture in a survey the intricate 
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decision-making processes within the institutions and study programmes, in which implementation 
(and anticipation!) of the qa process is mingled with other national and local policy initiatives, 
changes in the funding formulas for institutions and individual students and changes in labour 
market demands and opportunities. 
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